Controlled Experimentation in Naturalistic Mobile Settings

Purpose: Performing controlled user experiments on small devices in naturalistic mobile settings has always proved to be a difficult undertaking for many Human Factors researchers. Difficulties exist, not least, because mimicking natural small device usage suffers from a lack of unobtrusive data to guide experimental design, and then validate that the experiment is proceeding naturally. Methods: Here we use observational data to derive a set of protocols and a simple checklist of validations which can be built into the design of any controlled experiment focused on the user interface of a small device. These, have been used within a series of experimental designs to measure the utility and application of experimental software. Results: The key-point is the validation checks – based on the observed behaviour of 400 mobile users – to ratify that a controlled experiment is being perceived as natural by the user. Conclusions: While the design of the experimental route which the user follows is a major factor in the experimental setup, without check validations based on unobtrusive observed data there can be no cerThis work was undertaken as part of the RIAM project funded by the EPSRC (EP/E002218/1) and conducted in the Web Ergonomics Lab part of the School of Computer Science at the University of Manchester (UK). About: http://wel.cs.manchester.ac.uk Simon Harper University of Manchester, School of Computer Science Manchester, M13 9PL, UK. [email protected] http://simon.harper.name